Skip to content

Conversation

@frankmcsherry
Copy link
Member

An alternate approach to the problem identified in #702. This version avoids some other quadratic behavior (repeatedly draining a prefix of a list) through some peekable iterator shenanigans. It might be worth writing the iterator combinator that does this manually, just for clarity, but we can discuss. It was surprisingly grotty to get what felt like pretty simple behavior, but .. maybe that's a me problem.

Up next, putting together a test that reveals the bad behavior, and its absence.

@antiguru

@frankmcsherry
Copy link
Member Author

I have totally failed to cause a timely program to have lots and lots of capabilities at the same time. Sending 1M messages through an exchange among three workers results in ~3 updates recorded in reachability, takes ~1s on master. I can imagine it can happen, but various smarts seem to be preventing it from happening in the easy cases.

Copy link
Member

@antiguru antiguru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes sense!

I tried a different variant of my approach using Vec::extract_if, but since this API isn't available for SmallVec, we're stuck converting between SmallVec and Vec all the time. This isn't too bad unless the number of elements is less than or equal to the inline capacity, which requires a new heap allocation due to constraints of how SmallVec is implemented.

A second thought is that we could convert the worklist to a BTreeMap<Location, BinaryHeap<(Time, Diff)>>, which would allow us to work through per-location updates in bulk, at the expense of having to iterate the mapping.

@frankmcsherry frankmcsherry merged commit d180a31 into TimelyDataflow:master Aug 28, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants